Open
Conversation
JunAishima
requested changes
Aug 8, 2025
Collaborator
JunAishima
left a comment
JunAishima
left a comment
There was a problem hiding this comment.
suggestions to help future reviewers evaluate the autoproc processing system, as well as preventing having to read the autoproc blacklist more than once during a server lifecycle
| setBlConfig("auto_proc_lock", False) | ||
| start_in_proc = None | ||
| for proc_num in ['uranus-cpu044', 'uranus-cpu041', 'uranus-cpu021']: | ||
| if not getBlConfig(proc_num): |
Collaborator
There was a problem hiding this comment.
Looks like the BlConfig flag for proc_num is set, then cleared in the autoproc.py file in lsdc_processing.
All the more reason to make sure the changes related to autoproc are put into a PR for lsdc_processing!
| os.system(comm_s) | ||
|
|
||
| if runAutoProc: | ||
| setBlConfig("auto_proc_lock", True) |
Collaborator
There was a problem hiding this comment.
Did you find that this locking was necessary to prevent breaking the queue?
| node = getBlConfig(nodeName) | ||
| dimpleNode = getBlConfig("dimpleNode") | ||
| if (daq_utils.detector_id == "EIGER-16"): | ||
| with open(f"{os.environ['CONFIGDIR']}/autoproc_blacklist.json") as autoproc_blacklist_file: |
Collaborator
There was a problem hiding this comment.
could this be moved somewhere so that the file does not have to be json.loaded every time the check is performed?
vshekar
force-pushed
the
auto-autoproc
branch
from
71dd8f6 to
454ec58
Compare
vshekar
force-pushed
the
auto-autoproc
branch
from
454ec58 to
bbbbb33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial implementation of auto-autoproc
Using the conftrak to keep track of which collections that need to be processed by autoproc. LSDC server first checks if autoproc should be run on a proposal. If it should, then run it right after starting fastdp.
When running autoproc, it will first add the collection to a queue. Then it will start a processing script from a list of assigned processors. The processing script will separately pull the first collection from the queue and will keep running until the queue is empty. The processing script is in lsdc_processing
The only role of this PR is to: